Independence

Law of Total Probability

Exercise 1 (Countably infinite coin tosses). Consider a sequence of coin tosses, such that the sample space is \(\Omega = \left\{H,T\right\}^\mathbb{N}\). For set of outcomes \(E_n \triangleq \left\{\omega \in \Omega: \omega_n = H\right\}\), we consider an event space generated by \(\mathscr{F}\triangleq \sigma(\left\{E_n: n\in \mathbb{N}\right\})\). Let \(\mathscr{F}_n\) be the event space generate by the first \(n\) coin tosses, i.e. \(\mathscr{F}_n \triangleq \sigma(\left\{E_i: i \in [n]\right\})\). Let \(A_n\) be the set of outcomes corresponding to at least one head in first \(n\) outcomes \(A_n \triangleq \left\{\omega \in \Omega: \omega_i = H \text{ for some } i \in [n]\right\} = \cup_{i=1}^nE_i \in \mathscr{F},\) and \(B_n\) be the set of outcomes corresponding to first head at the \(n\)th outcome \(B_n \triangleq \left\{\omega \in \Omega: \omega_1 = \dots = \omega_{n-1}= T, \omega = H\right\} = \cap_{i=1}^{n-1}E_i^c\cap E_n \in \mathscr{F}.\)

1. Show that \(\mathscr{F}= \sigma(\left\{\mathscr{F}_n: n \in \mathbb{N}\right\})\).

2. Show that \(\sigma(\left\{A_n: n \in \mathbb{N}\right\}) \subseteq \mathscr{F}\) and \(\sigma(\left\{B_n: n\in \mathbb{N}\right\}) \subseteq \mathscr{F}\).

Theorem 2 (Law of total probability). For a probability space \((\Omega, \mathscr{F}, P)\), consider a sequence of events \(B\in \mathscr{F}^\mathbb{N}\) that partitions the sample space \(\Omega\), i.e. \(B_m \cap B_n = \emptyset\) for all \(m \neq n\), and \(\cup_{n \in \mathbb{N}}B_n = \Omega\). Then, for any event \(A \in \mathscr{F}\), we have \[\begin{equation*} P(A) = \sum_{n \in \mathbb{N}}P(A \cap B_n). \end{equation*}\]

Proof. Proof. We can expand any event \(A \in \mathscr{F}\) in terms of any partition \(B\) of the sample space \(\Omega\) as \[\begin{equation*} A = A \cap \Omega = A \cap (\cup_{n \in \mathbb{N}}B_n) = \cup_{n \in \mathbb{N}}(A \cap B_n). \end{equation*}\] From the mutual disjointness of the events \(B \in \mathscr{F}^\mathbb{N}\), it follows that the sequence \((A \cap B_n \in \mathscr{F}: n \in \mathbb{N})\) is mutually disjoint. The result follows from the countable additivity of probability of disjoint events. ◻

Example 3 (Countably infinite coin tosses). Consider the sample space \(\Omega = \left\{H,T\right\}^\mathbb{N}\) and event space \(\mathscr{F}\) generated by sequence \(E \in \mathscr{F}^\mathbb{N}\) defined in Exercise 1. We observe that any event \(A \in \mathscr{F}_n\) can be written as \[\begin{equation*} A = \cup_{\omega \in A}\left\{\omega\right\} = \cup_{\omega \in A}\cap_{i=1}^n(\left\{\omega \in E_i\right\}\cup\left\{\omega \notin E_i\right\}). \end{equation*}\]

Independence

Definition 4 (Independence of events). For a probability space \((\Omega, \mathscr{F}, P)\), a family of events \(A \in \mathscr{F}^I\) is said to be independent, if for any finite set \(F \subseteq I\), we have \[\begin{equation*} P(\cap_{i \in F}A_i) = \prod_{i \in F}P(A_i). \end{equation*}\]

Remark 1. The certain event \(\Omega\) and the impossible event \(\emptyset\) are always independent to every event \(A \in \mathscr{F}\).

Example 5 (Two coin tosses). Consider two coin tosses, such that the sample space is \(\Omega = \left\{HH, HT, TH, TT\right\}\), and the event space is \(\mathscr{F}= \mathcal{P}(\Omega)\). It suffices to define a probability function \(P: \mathscr{F}\to [0,1]\) on the sample space. We define one such probability function \(P\), such that \[\begin{equation*} P(\left\{HH\right\}) = P(\left\{HT\right\}) = P(\left\{TH\right\}) =P(\left\{TT\right\}) = \frac{1}{4}. \end{equation*}\] Let event \(E_1 \triangleq \left\{HH, HT\right\}\) and \(E_2 \triangleq \left\{HH, TH\right\}\) correspond to getting a head on the first or the second toss respectively.

From the defined probability function, we obtain the probability of getting a tail on the first or the second toss is \(\frac{1}{2}\), and identical to the probability of getting a head on the first or the second toss. That is, \(P(E_1) = P(E_2) = \frac{1}{2}\) and the intersecting event \(E_1\cap E_2 = \left\{HH\right\}\) with the probability \(P(E_1 \cap E_2) = \frac{1}{4}\). That is, for events \(E_1, E_2 \in \mathscr{F}\), we have \[\begin{equation*} P(E_1 \cap E_2) = P(E_1)P(E_2). \end{equation*}\] That is, events \(E_1\) and \(E_2\) are independent.

Example 6 (Countably infinite coin tosses). Consider the outcome space \(\Omega = \left\{H,T\right\}^\mathbb{N}\) and event space \(\mathscr{F}\) generated by the sequence \(E\) defined in Exercise 1. We define a probability function \(P:\mathscr{F}\to [0,1]\) by \(P(\cap_{i\in F}E_i)= p^{\left\lvert F\right\rvert}\) for any finite subset \(F \subseteq \mathbb{N}\). By definition, \(E \in \mathscr{F}^\mathbb{N}\) is a sequence of independent events. Consider \(A, B \in \mathscr{F}^\mathbb{N}\), where \(A_n \triangleq \cup_{i=1}^nE_i\) and \(B_n \triangleq \cap_{i=1}^{n-1}E_i^c\cap E_n \in \mathscr{F}\) for all \(n\in\mathbb{N}\). It follows that \(P(A_n) = 1-(1-p)^n\) and \(P(B_n) = p(1-p)^{n-1}\) for \(n \in \mathbb{N}\).

For any \(\omega \in \Omega\), we can define the number of heads in first \(n\) trials by \(k_n(\omega) \triangleq\sum_{i=1}^n\mathbbm{1}_{\left\{H\right\}}(\omega_i) = \sum_{i=1}^n\mathbbm{1}_{\left\{\omega \in E_i\right\}}\). For any general event \(A \in \mathscr{F}_n = \sigma(\left\{E_i: i \in [n]\right\})\), we can write \[\begin{equation*} P(A) = \sum_{\omega \in A} \prod_{i=1}^n\Big[P\left\{\omega \in E_i\right\} + P\left\{\omega \in E_i^c\right\}\Big] = \sum_{\omega \in A}p^{k_n(\omega)}(1-p)^{n-k_n(\omega)}. \end{equation*}\]

Example 7 (Counter example). Consider a probability space \((\Omega, \mathscr{F}, P)\) and the events \(A_1, A_2, A_3 \in \mathscr{F}\). The condition \(P(A_1\cap A_2\cap A_3) = P(A_1)P(A_2)P(A_3)\) is not sufficient to guarantee independence of the three events. In particular, we see that if

2 &P(A_1A_2A_3) = P(A_1)P(A_2)P(A_3),&&P(A_1A_2A_3^c) P(A_1)P(A_2)P(A_3^c),

then \(P(A_1\cap A_2) = P(A_1\cap A_2\cap A_3) + P(A_1\cap A_2\cap A_3^c) \neq P(A_1)P(A_2)\).

Definition 8. A family of collections of events \((\mathscr{A}_i \subseteq \mathscr{F}: i \in I)\) is called independent, if for any finite set \(F \subseteq I\) and \(A_i \in \mathscr{A}_i\) for all \(i \in F\), we have \[\begin{equation*} P(\cap_{i \in F}A_i) = \prod_{i \in F}P(A_i). \end{equation*}\]

Conditional Probability

Consider \(N\) trials of a random experiment over an outcome space \(\Omega\) and an event space \(\mathscr{F}\). Let \(\omega_n \in \Omega\) denote the outcome of the experiment of the \(n\)th trial. Consider two events \(A, B \in \mathscr{F}\) and denote the number of times event \(A\) and event \(B\) occurs by \(N(A)\) and \(N(B)\) respectively. We denote the number of times both events \(A\) and \(B\) occurred by \(N(A\cap B)\). Then, we can write these numbers in terms of indicator functions as

3 &N(A) = _n=1^N_{_n A}, &&N(B) = _n=1^N_{_n B}, &&N(AB) = _n=1^N_{_n A B}.

We denote the relative frequency of events \(A, B, A\cap B\) in \(N\) trials by \(\frac{N(A)}{N}, \frac{N(B)}{N}, \frac{N(A\cap B)}{N}\) respectively. We can find the relative frequency of events \(A\), on the trials where \(B\) occurred as \[\begin{equation*} \frac{\frac{N(A\cap B)}{N}}{\frac{N(B)}{N}} = \frac{N(A \cap B)}{N(B)}. \end{equation*}\] Inspired by the relative frequency, we define the conditional probability function conditioned on events.

Definition 9. Fix an event \(B \in \mathscr{F}\) such that \(P(B) > 0\), we can define the conditional probability \(P(\cdot | B): \mathscr{F}\to [0,1]\) of any event \(A \in \mathscr{F}\) conditioned on the event \(B\) as \[\begin{equation*} P(A|B) = \frac{P(A\cap B)}{P(B)}. \end{equation*}\]

Lemma 10 (Conditional probability). For any event \(B \in \mathscr{F}\) such that \(P(B) > 0\), the conditional probability \(P(\cdot | B): \mathscr{F}\to [0,1]\) is a probability measure on space \((\Omega, \mathscr{F})\).

Proof. Proof. We will show that the conditional probability satisfies all three axioms of a probability measure.

1. For all events \(A \in \mathscr{F}\), we have \(P(A|B) \geqslant 0\) since \(P(A \cap B) \geqslant 0\).

2. For an infinite sequence of mutually disjoint events \((A_i \in \mathscr{F}: i \in \mathbb{N})\) such that \(A_i \cap A_j = \emptyset\) for all \(i \neq j\), we have \(P(\cup_{i \in \mathbb{N}}A_i|B) = \sum_{i \in \mathbb{N}}P(A_i|B)\). This follows from disjointness of the sequence \((A_i\cap B \in \mathscr{F}: i \in \mathbb{N})\).

3. Since \(\Omega \cap B = B\), we have \(P(\Omega |B) = 1\).

 ◻

Remark 2. For two independent events \(A, B \in \mathscr{F}\) such that \(P(A\cap B) > 0\), we have \(P(A|B) = P(A)\) and \(P(B|A) = P(B)\). If either \(P(A) = 0\) or \(P(B) = 0\), then \(P(A \cap B) = 0\).

Remark 3. For any partition \(B\) of the sample space \(\Omega\), if \(P(B_n) > 0\) for all \(n \in \mathbb{N}\), then from the law of total probability and the definition of conditional probability, we have \[\begin{equation*} P(A) = \sum_{n \in \mathbb{N}}P(A|B_n)P(B_n). \end{equation*}\]

Conditional Independence

Definition 11 (Conditional independence of events). For a probability space \((\Omega, \mathscr{F}, P)\), a family of events \(A \in \mathscr{F}^I\) is said to be conditionally independent given an event \(C \in \mathscr{F}\) such that \(P(C) > 0\), if for any finite set \(F \subseteq I\), we have \[\begin{equation*} P(\cap_{i \in F}A_i|C) = \prod_{i \in F}P(A_i|C). \end{equation*}\]

Remark 4. Let \(C \in \mathscr{F}\) be an event such that \(P(C) > 0\). Two events \(A, B \in \mathscr{F}\) are said to be conditionally independent given event \(C\), if \[\begin{equation*} P(A \cap B|C) = P(A|C) P(B|C). \end{equation*}\] If the event \(C = \Omega\), it implies that \(A, B\) are independent events.

Remark 5. Two events may be independent, but not conditionally independent and vice versa.

Example 12. Consider two independent events \(A, B \in \mathscr{F}\) such that \(P(A\cap B) > 0\) and \(P(A \cup B) < 1\). Then the events \(A\) and \(B\) are not conditionally independent given \(A\cup B\). To see this, we observe that \[\begin{equation*} P(A \cap B | A\cup B) = \frac{P( (A \cap B)\cap (A \cup B))}{P(A\cup B)} = \frac{P(A \cap B)}{P(A\cup B)} = \frac{P(A)P(B)}{P(A \cup B)} = {P(A|A \cup B)}{P(B)}. \end{equation*}\] We further observe that \(P(B|A\cup B) = \frac{P(B)}{P(A\cup B)} \neq P(B)\) and hence \(P(A \cap B | A\cup B) \neq P(A|A\cup B)P(B|A\cup B)\).

Example 13. Consider two non-independent events \(A, B \in \mathscr{F}\) such that \(P(A) > 0\). Then the events \(A\) and \(B\) are conditionally independent given \(A\). To see this, we observe that \[\begin{equation*} P(A \cap B | A) = \frac{P(A \cap B)}{P(A)} = P(B|A)P(A|A). \end{equation*}\]